SMTP CPU Spike 50-100%

HCL Notes/Domino 8.5 Forum (includes Notes Traveler)

Subject:

Hi Steve and Mark,

Yes this issue has been reported to IBM and we have reproduced this issue in house on Windows and AIX servers. We are currently working on a fix for this issue and we have documented this Under Technote:

http://www-01.ibm.com/support/docview.wss?uid=swg21445280

and it is being tracked under the following new SPR#LMIL88Q62X

When You run an NSD it may take a long time however you will see:

On Windows:

############################################################
### thread 4/6: [ nSMTP: 0cc0: 17a8]
### FP=0x150dd7e8, PC=0x600013e6, SP=0x150dd7ac
### stkbase=0x150e0000, total stksize=262144, used stksize=10324
############################################################
@[ 1] 0x600013e6 nnotes.OSUnlockSem@4+54 (15bb72b4)
@[ 2] 0x601df473 nnotes.UnlockMemHandle@4+19 (15bb728c)
@[ 3] 0x60002c31 nnotes.OSUnlockObject@4+129 (611e9090)
@[ 4] 0x6001b68c nnotes.NSFAddToNameLookupTable@12+268 (15c22208,1ed,611e7718)
@[ 5] 0x6001dd2a nnotes.ItemAppendByBLOCKID@32+570 (150dd8b4,4,60eebc54,4,0,150dd870,f10f10,ffffffff)
@[ 6] 0x60048ecd nnotes.NSFItemAppendByBLOCKID@32+93 (59,3,60eebc54,15c22110,0,150dd8a0,f10f10,ffffffff)
@[ 7] 0x600522e0 nnotes.NSFItemAppend@28+160 (59,3,60eebc54,4,0,150dd8dc,f10f10)
@[ 8] 0x60358756 nnotes.NSFMimePartAppendTG@36+182 (59,60eebc54,4,2,0,150dd928,f10f10,ffffffff,0)
@[ 9] 0x60351e29 nnotes.MIMEItemizeMessageBodyItemAppend+2393 (150deb30,59,60eebc54,4)
@[10] 0x603527da nnotes.MIMEItemizeMessageBody+1162 (150deb30,59,60eebc54,4)
@[11] 0x603522d4 nnotes.MIMEItemizeMessage+804 (150deb30,59,60eebc54,4)
@[12] 0x603525b7 nnotes.MIMEItemizeMessageBody+615 (150deb30,59,60eebc54,4)
@[13] 0x603522d4 nnotes.MIMEItemizeMessage+804 (150deb30,59,60eebc54,4)
@[14] 0x603525b7 nnotes.MIMEItemizeMessageBody+615 (150deb30,59,60eebc54,4)
@[15] 0x603522d4 nnotes.MIMEItemizeMessage+804 (150deb30,59,60eebc54,4)
@[16] 0x60352a74 nnotes.MIMEItemizeUsingCallbackExt@32+372 (59,60eebc54,4,7,0,150deb40,f10f10,ffffffff)
@[17] 0x60352cad nnotes.MIMEItemizeStreamExt@28+93 (59,0,4,7,0,150dee78,f10f10)
@[18] 0x603531a2 nnotes.MIMEItemizeStream@24+34 (59,0,0,7,0,150dee9c)
@[19] 0x602b24ed nnotes.CIMsgImport::Import+221 (14f513b8,397030,59,3)
@[20] 0x004136d3 nSMTP.CSMTPProtocol::SubmitMessage+467 (14f513b8,150dfc6a,2,14648018)
@[21] 0x0041192f nSMTP.CSMTPProtocol::CommandDATA+1407 (8018,2,14648018,0)
@[22] 0x00412bb6 nSMTP.CSMTPProtocol::StateConnected+1158 (14648018,390000,397010,397008)
@[23] 0x00412ecd nSMTP.CSMTPProtocol::Run+429 (14648018,128de6c,43bf68,128de6c)
@[24] 0x0042183e nSMTP.CBaseTask::StateMachine+398 (397008,128de6c,43bf68,128de88)
@[25] 0x00402fe3 nSMTP.CSMTPSrv::OnConnect+211 (43bf68,128de88,13d80001,128de6c)
@[26] 0x004173a6 nSMTP.CIServ::ServerTaskProtocolMachine+262 (43bf68,128de6c,1232ee8,12357f0)
@[27] 0x0041bd9a nSMTP.CIServ::ServerTaskIOCP+922 (43bf68,0,6015584f,0)
@[28] 0x0041c71d nSMTP.ServerThread@4+29 (0)
[29] 0x77e6482f kernel32.GetModuleHandleA+223 (601557a0,0,0,dddd04d2)

also we saw this stack as well:

############################################################
### thread 7/17: [ nSMTP: 052c: 0cc8]
### FP=0x1978c054, PC=0x600013e6, SP=0x1978c018
### stkbase=0x19790000, total stksize=262144, used stksize=16360
############################################################
@[ 1] 0x600013e6 nnotes.OSUnlockSem@4+54 (18ff3808)
@[ 2] 0x601df473 nnotes.UnlockMemHandle@4+19 (18ff37e0)
@[ 3] 0x601df5cf nnotes.OSLockObjectExt@8+143 (611e9090,611e9142)
@[ 4] 0x60002b7e nnotes.OSLockObject@4+14 (94)
@[ 5] 0x6001b694 nnotes.NSFAddToNameLookupTable@12+276 (a40a08,1ef,611ed988)
@[ 6] 0x6001dd2a nnotes.ItemAppendByBLOCKID@32+570 (1978c138,4,60eebc54,4,0,1978c0f4,f10f10,ffffffff)
@[ 7] 0x60048ecd nnotes.NSFItemAppendByBLOCKID@32+93 (83,3,60eebc54,a40910,0,1978c124,f10f10,ffffffff)
@[ 8] 0x600522e0 nnotes.NSFItemAppend@28+160 (83,3,60eebc54,4,0,1978c160,f10f10)
@[ 9] 0x60358756 nnotes.NSFMimePartAppendTG@36+182 (83,60eebc54,4,2,0,1978c1ac,f10f10,ffffffff,0)
@[10] 0x60351e29 nnotes.MIMEItemizeMessageBodyItemAppend+2393 (1978d3b4,83,60eebc54,4)
@[11] 0x603527da nnotes.MIMEItemizeMessageBody+1162 (1978d3b4,83,60eebc54,4)
@[12] 0x603522d4 nnotes.MIMEItemizeMessage+804 (1978d3b4,83,60eebc54,4)
@[13] 0x603525b7 nnotes.MIMEItemizeMessageBody+615 (1978d3b4,83,60eebc54,4)
@[14] 0x603522d4 nnotes.MIMEItemizeMessage+804 (1978d3b4,83,60eebc54,4)
@[15] 0x603525b7 nnotes.MIMEItemizeMessageBody+615 (1978d3b4,83,60eebc54,4)
@[16] 0x603522d4 nnotes.MIMEItemizeMessage+804 (1978d3b4,83,60eebc54,4)
@[17] 0x60352a74 nnotes.MIMEItemizeUsingCallbackExt@32+372 (83,60eebc54,4,13,0,1978d3c4,f10f10,ffffffff)
@[18] 0x60352cad nnotes.MIMEItemizeStreamExt@28+93 (83,60eebc54,4,13,0,1978d6fc,f10f10)
@[19] 0x603531a2 nnotes.MIMEItemizeStream@24+34 (83,60eebc54,4,13,0,1978d720)
@[20] 0x602afd79 nnotes.CIMsgImport::ImportConvertDeliveryStatus+505 (19891498,83,1,1978f000)
@[21] 0x602b0ba8 nnotes.CIMsgImport::ImportConvertHeaders+216 (19891498,c736d0,83,1978f000)
@[22] 0x602b2587 nnotes.CIMsgImport::Import+375 (19891498,c736d0,83,3)
@[23] 0x004136d3 nSMTP.CSMTPProtocol::SubmitMessage+467 (19891498,1978fc6a,2,19cba018)
@[24] 0x0041192f nSMTP.CSMTPProtocol::CommandDATA+1407 (a018,2,19cba018,656e6968)
@[25] 0x00412bb6 nSMTP.CSMTPProtocol::StateConnected+1158 (19cba018,c70000,c736b0,c736a8)
@[26] 0x00412ecd nSMTP.CSMTPProtocol::Run+429 (19cba018,12ddd64,43bf68,12ddd64)
@[27] 0x0042183e nSMTP.CBaseTask::StateMachine+398 (c736a8,12ddd64,43bf68,12ddd80)
@[28] 0x00402fe3 nSMTP.CSMTPSrv::OnConnect+211 (43bf68,12ddd80,12240001,12ddd64)
@[29] 0x004173a6 nSMTP.CIServ::ServerTaskProtocolMachine+262 (43bf68,12ddd64,1282ee8,12857f0)
@[30] 0x0041bd9a nSMTP.CIServ::ServerTaskIOCP+922 (43bf68,0,6015584f,0)
@[31] 0x0041c71d nSMTP.ServerThread@4+29 (0)
[32] 0x77e6482f kernel32.GetModuleHandleA+223 (0,0,0,0)

On AIX we see:

Thread consuming CPU
###################################
###### thread 4/6 :: smtp, pid=196710, tid=2089079, ptid=1029) ######
###################################
[1] 0x0000000000003438 _check_lock() + 0x18
[2] 0x0000000110ee5c8c ????????(??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??) + ??
[3] 0x09000000006ede70 UnlockHandle(0x1100ac798) + 0x1c
[4] 0x09000000006ef054 OSLockObjectExt(0x1d0000001d, 0x0) + 0xd4
[5] 0x09000000006ef1dc OSLockObject(0x1d0000001d) + 0x1c
[6] 0x0900000000b4fd6c NSFAddToNameLookupTable(0x11019df08, 0x2deb38000a) + 0x218
[7] 0x0900000000b665e8 ItemAppendByBLOCKID(0x110ee6070, 0x3000000000003, 0x9000000028cddd8, 0x4000000000004, 0x2deaf0003a, 0x3c0000003c, 0x0) + 0x254
[8] 0x0900000000b67430 NSFItemAppendByBLOCKID(0xa0000000a, 0x3000000000003, 0x9000000028cddd8, 0x4000000000004, 0x2deaf0003a, 0x3c0000003c, 0x0) + 0xec
[9] 0x0900000000b67d3c NSFItemAppend(0xa0000000a, 0x3000000000003, 0x9000000028cddd8, 0x4000000000004, 0x19000000000019, 0x11003cce0, 0x3a0000003a) + 0x134
[10] 0x0900000000ea9b68 NSFMimePartAppendTG(0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x2000000000002, 0x100000001, 0x110ee6408, 0x26000000000026, 0x26000000000026) + 0x170
[11] 0x090000000102f47c MIMEItemizeMessageBodyItemAppend(tagAttachContext*,unsigned int,char*,unsigned short,unsigned int,tagMIMEPartType,int,int,tagMimeWellKnownSymbolId,tagMimeWellKnownSymbolId,tagMimeWellKnownSymbolId,tagMimeWellKnownSymbolId,CMemStr&,CMemStr&,CMemStr&,CStream*,CMemStr&)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1200000012, 0x200000002, 0x100000001, 0x100000001) + 0xe48
[12] 0x090000000102fa7c MIMEItemizeMessageBody(tagAttachContext*,unsigned int,char*,unsigned short,unsigned int,CMIMEReadBuffer&,CMimeParse&)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1200000012, 0x110ee84d0, 0x110ee7200) + 0x364
[13] 0x090000000102d108 MIMEItemizeMessage(tagAttachContext*,unsigned int,char*,unsigned short,unsigned int,CMIMEReadBuffer&,CMimeParse&,unsigned int*)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1200000012, 0x110ee84d0, 0x110ee7200, 0x0) + 0x4c8
[14] 0x090000000102f988 MIMEItemizeMessageBody(tagAttachContext*,unsigned int,char*,unsigned short,unsigned int,CMIMEReadBuffer&,CMimeParse&)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1200000012, 0x110ee84d0, 0x110ee7b90) + 0x270
[15] 0x090000000102d108 MIMEItemizeMessage(tagAttachContext*,unsigned int,char*,unsigned short,unsigned int,CMIMEReadBuffer&,CMimeParse&,unsigned int*)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1200000012, 0x110ee84d0, 0x110ee7b90, 0x0) + 0x4c8
[16] 0x090000000102f988 MIMEItemizeMessageBody(tagAttachContext*,unsigned int,char*,unsigned short,unsigned int,CMIMEReadBuffer&,CMimeParse&)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1200000013, 0x110ee84d0, 0x110ee8510) + 0x270
[17] 0x090000000102d108 MIMEItemizeMessage(tagAttachContext*,unsigned int,char*,unsigned short,unsigned int,CMIMEReadBuffer&,CMimeParse&,unsigned int*)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1300000013, 0x110ee84d0, 0x110ee8510, 0x0) + 0x4c8
[18] 0x090000000102c2e8 MIMEItemizeUsingCallbackExt(0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1300000013, 0x9001000a0781240, 0x110ee88b0, 0x0, 0x0) + 0x178
[19] 0x090000000102bb04 MIMEItemizeStreamExt(0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1300000013, 0x110ee9760, 0x0, 0x0) + 0xc4
[20] 0x090000000102ba04 MIMEItemizeStream(0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1300000013, 0x110ee9760, 0x0) + 0x44
[21] 0x0900000001038a7c CIMsgImport::ImportConvertDeliveryStatus(unsigned int,int,unsigned int*)(0x1100d2cf8, 0xa0000000a, 0x100000001, 0x110eeac44) + 0x364
[22] 0x0900000001035880 CIMsgImport::ImportConvertHeaders(CStream&,unsigned int,unsigned int*,CONVERSION_CONTROLS*,int)(0x1100d2cf8, 0x1113161e8, 0xa0000000a, 0x110eeac44, 0x700000004ee64a0, 0x100000001) + 0x11f0
[23] 0x09000000010343c4 CIMsgImport::Import(CStream&,unsigned int,unsigned short,unsigned int,CONVERSION_CONTROLS*)(0x1100d2cf8, 0x1113161e8, 0xa0000000a, 0x3000000000003, 0x2700000027, 0x700000004ee64a0) + 0x204
[24] 0x000000010002f7b8 CSMTPProtocol::SubmitMessage(unsigned short*)(0x1100eb638, 0x110eebcc0) + 0x308
[25] 0x000000010002a418 CSMTPProtocol::CommandDATA()(0x1100eb638) + 0x138
[26] 0x00000001000267c0 CSMTPProtocol::StateConnected()(0x1100eb638) + 0x25c
[27] 0x0000000100024630 CSMTPProtocol::Run()(0x1100eb638) + 0x29c
[28] 0x000000010001bd78 CBaseTask::StateMachine()(0x1113161b0) + 0x25b0
[29] 0x00000001000049f0 CSMTPSrv::OnConnect(SESSIONID&,unsigned int,INETTASK*)(0x1100005c8, 0x1101e9068, 0x110d0000110d0, 0x1101e9038) + 0x98
[30] 0x0000000100013f10 CIServ::ServerTaskProtocolMachine(INETTASK*)(0x1100005c8, 0x1101e9038) + 0x5c8
[31] 0x00000001000122c4 CIServ::ServerTaskIOCP(int)(0x1100005c8, 0x100000001) + 0x7ac
[32] 0x0000000100011a8c ServerThread(0x0) + 0x30
[33] 0x09000000006b6900 ThreadWrapper(0x0) + 0x118
[34] 0x09000000004a34f4 _pthread_body(??) + 0xdc

Feedback response number DADS88TKXE created by ~Evelyn Zenresamanikle on 08/30/2010